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About this talk 

1. Intro 

2. How we work 

3. Analytics tools 

4. Project examples 

5. Monitoring game health 

6. Recommendations 
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Intro - About me 

• RuneScape studio head @ Jagex 

• Worked in game design for 17 years 

• 1998 - Bullfrog / Electronic Arts 

• 2001 - Sony 

• 2006 - Climax 

• 2011 - Jagex 
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Intro - About RuneScape 

• Traditional fantasy MMORPG 

• Launched 2001 - updated weekly 

• F2P Hybrid: Free access, optional subscription and MTX 

• 160 development and publishing staff 

• Franchise: Old School, Chronicle, new MMO and more to come 
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Intro - About Our Players 

• Mostly English speaking males 16-26 

• Over half have played 6+ years 

• 90 minute average session 

• Active video & streaming community 
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Intro - RuneScape Key Stats 


• Revenue to date: 

• Annual revenue: 

• New users: 

• Daily actives: 

• Monthly actives: 

• Subscribers: 

• Concurrent users: 


$750M~ 

$75M+ 

20k/day 

500-600k 

2M 

600k+ 

70k av, 100k peak 
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Intro - Why use game analytics? 

• More information for making decisions 

• Provides accuracy and scale 

• Especially useful for refining live games 

• Minimize guesswork in the design process 

•BUT... 

• Analytics is not a substitute for creativity or good design! 
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Intro - RuneScape as a case study 

RuneScape has demanding requirements: 

• Large game with 10,000 hours content 

• Non-linear, open world and player-driven economy 

• Weekly content additions & changes 

• Game analytics started 10 years after launch 
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How We Work - Data services at Jagex 

1. Business Intelligence - Data Warehousing, Reporting, Data Science 

2. Insight - Customer & market research 

3. Embedded Analytics - RuneScape, Block N Load, Digital Marketing 

• 18 people total 

• Several years of strategic investment 
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How We Work - RS Game Analytics 














\ * 
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Jason - Game analyst - game content, economy, balancing 

• Level 2595 (max level) - 358 days play time 


• Matt - Commercial analyst - monetisation, funnel, game health 

* Level 2444 - 160 days play time 


Simon - Commercial analyst - KPIs, forecasts, payments, risk 


Level 3 - Noob! 


Joel - Analytics Lead 

* Level 2560 - 190 days play time 
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How We Work - RS Game Analytics 

Ways of working: 

• Automated reports 

• Self-service tools 

• Post project analysis 

• Ad-hoc data requests 

• Pro-active reports 

• Monthly reports 

• Player facing data 
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How We Work - RS Game Analytics 

• Automated reporting e.g. Product KPIs, item sales 


RuneScape Targets: Retention 2015 


Right Now: Member Retention 


above target 


Data refreshed to 07 May 2015. 


Daily retention against averaged target 
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How We Work - RS Game Analytics 


• Self Service Tools e.g. Player Profile Suite 
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How We Work - RS Game Analytics 


Post project analysis e.g. weekly content releases 


RuneScape Analytics - Heart of Stone 

Quest Uu.'Khtfd M/li/JOM. Data taken bom 24/U/WU to 0&/1J/X114 


Farlmpation and completion rates n «*•*•! by hour 


Heart 


0VU/20S4 


SU|e completion of partHipatrryi players 


ia.ii.mi4 


1*11/3014 


Completion Skill Total vv. Combat Ml i 


Completion Combat n. Skill loUl Ml a 
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How We Work - RS Game Analytics 

• Ad-hoc data requests e.g. bespoke research or review 


20,000 


15.000 


10,000 


5,000 


RSM Well of goodwi ll Donations USD (5m = 1$ | lbond = 4.7$) 

HHpi HHHH HHHHHHH HHlHOa HHHHHHpl HTlHHHH Hrt 


Goins ■ Bond 


Superbowl impact on active players 

USA RuneScape CCU 

1 Week before Superbowl Sunday® 








Mod Porky gJagexPorky Feb 9 

I wouldn't want to fight against these; 
RuneScape's Top Boss Killers by killcount 


Araxxi xRegal 

Barrows Iambal89 

Chaos Elemental Pio Kildare 

Commander Zilyana Efb 

Commander Zilyana (Hard) Omid 
Corporeal Beast Bjoerun 

Dagannoth Kings x Auspnde 

General Graardor King DeJay 

General Graardor (Hard) Divine Eel 
Giant Mole Ben Has A 
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How We Work - RS Game Analytics 


Player facing data e.g. infographics and tweets 


l Minute in Game 


The world of Gietinor is ever changing 
During a single minute in RuneScape... 


TOP BOSS KILLERS 


BOSSES 

KILLED 


NEW 

PLAYERS 


More than 

307000,000 XP 


Join the adventure 
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How We Work - RS Game Analytics 


Monthly Reports 
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Tools - Analytics in the design process 

* Pre-production 

• What game feature should we work on next? 

• How can my designs be as successful as possible? 

• Post-launch \ 

• Did it work as I wanted and how can I make it better? 
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Tools - Pre-production Analytics 

“What game feature should we work on next?” 

• What problems need to be fixed? 

* Funnel analysis - Locate where player progress is stalling 

* Churn analysis - Identify what leavers have in common 
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Tools - Pre-production Analytics 

• - tutorial drop off points 


RuneScape Analytics - Ashdale Tutorial 

Launched 26/11/2013. Data taken from 26/11/2013 to 16/12/2013 



Participation and completion rates (to date) by day 


Quest Selections [Membership, combat and skill totals 


Notes 

ire those who completed 'AshdaleTutorial' within 24 hoi 
l. Players are able to repeat 'AshdaleTutorial' for fun ai 


las "X Pla\ 

Skipping to End From Any Stage - At any part of 'AshdaleTutorial' players can choose to skip (complete it) and return tc 
Taverly. For example, with no filters applied the stage completion chart shows the sum of those completing stage 1 

I to skip, the blue is those that went on to stage 3 and the drop in volume is those that are yet to do anything. 
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Skipping to End From Any Stage 


New Player - 1st Time 


Non-Skip Skip 

Player Tutorial Start Date 


16/12/2013 15/12/2013 14/12/2013 
13/12/2013 12/12/2013 11/12/2013 
10/12/2013 09/12/2013 08/12/2013 
07/12/2013 06/12/2013 05/12/2013 
04/12/2013 03/12/2013 02/12/2013 
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DATE OF QUEST ACCEPTANCE 

Started % Complete % Complete - 7d Rolling Avg 

Duration distribution (completing in less than 1 hour) 



MINUTES FROM QUEST START TO QUEST COMPLETION 


Completion - Skill Total vs. Combat Mix 
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■ Average Completion Rate 

• Mode Completion Time (Minutes) 

• Quickest Completion Time (Minutes) j 

• Largest stage drop off point (stage 

lookup to right) 

• Combat Band - Largest volume of 
players completing are in combat band ; 

• Skill Total Band - Largest volume of 
players completing are in skill total band j 

• Starting - average no. players per day ! 

• Completing - average no. players per j 


000-010 

0000-0200 

6947 

4269 

7284 


• Engaging - average no. players per day 

Stage completion of participating players 


89.98% of players who completed the 
previous stage went on to complete 
93.9% / 86529 of completions. 

99.88% / 92047 of completions. 

Could have started quest at any time 
previously, not just the date range chosen. 

Could have started quest at any time 
previously, not just the date range chosen. 



Completion - Combat vs. Skill Total Mix 


SKILL TOTAL BANDS 


■ 091-100 

■ 081-090 

■ 071-080 

■ 061-070 

■ 051-060 

■ 041-050 

■ 031-040 


COMBAT BANDS 


2401-2 

2201-2 

1801 

1601 

1401 

1201 
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0801 

0601-0 

0401-0 


1800 

1600 

1400 

1200 

1000 




Notes 

Account and Playthrough Type - New players are those who completed 'Ashdale Tutorial' within 24 hours of creation, or 
'Shadow Over Ashdale' with 7 days of creation. Players areableto repeat 'Ashdale Tutorial' for fun and will be recorded 
as "X Player - Repeat". 

Skipping to End From Any Stage - At any part of 'Ashdale Tutorial' players can choose to skip (complete it) and return to 
Taverly. For example, with no filters applied the stage completion chart shows the sum of those completing stage 1 
without skipping to the end (blue) and their outcome from stage 2. In the stage 2 column, the yellow is those that decided 
to skip, the blue is those that went on to stage 3 and the drop in volume is those that are yet to do anything. 


Quest (Select 1 Only) 

Ashdale Tutorial 

Dodgy 

False 

Account and Playthrough Type 


Skipping to End From Any Stage 


Shadow Over Ashdale 


Non-Skip 


Skip 


Player Tutorial Start Date 


Existing Player 

- 1st Time 

Existing Player 

- Repeat 


New Player - 1st Time 


New Player- Repeat 


1 19/12/2013 

1 18/12/2013 

1 17/12/2013 1 

16/12/2013 

15/12/2013 

14/12/2013 

13/12/2013 

12/12/2013 

11/12/2013 

10/12/2013 

09/12/2013 

08/12/2013 

07/12/2013 

06/12/2013 

05/12/2013 

04/12/2013 

03/12/2013 

02/12/2013 


Membership At 1st Tutorial Stage 


Skill Total Band At 1st Tutorial Stage 

0000-0200 0201-0400 


F2P P2P 

Combat Band At 1st Tutorial Stage 


000-010 011-020 

021-030 031-040 

041-050 051-060 
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1401-1600 
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Region 


Africa 

Americas 

Asia 

Europe 

Oceania 

Other 


Country 


Afghanistan 
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Angola 
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Anonymous Proxy 

1 Antarctica 


RuneScape Analytics - Ashdale Tutorial 

Launched 26/11/2013. Data taken from 26/11/2013 to 16/12/2013 


Participation and completion rates (to date ) by day 


Quest Selections [Membership, combat and skill totals are recorded at quest start stage] 
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DATE OF QUEST ACCEPTANCE 

Started % Complete % Complete - 7d Rolling Avg 


Duration distribution (completing in less than 1 hour) 
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Completion - Skill Total vs. Combat Mix 
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COMBAT 
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■ 121-130 

■ 111-120 
■ 101-110 

■ 091-100 

■ 081-090 

■ 071-080 

■ 061-070 
■'051-060 
” 041-050 

■ 031-040 


Analysis 

j Fact | 

1 Figure | 

Notes j 

• Average Completion Rate 

63.17% 


• Mode Completion Time (Minutes) 

15 


• Quickest Completion Time (Minutes) 

0 


• Largest stage drop off point (stage 
lookup to right) 

3 

89.98% of players who completed the 
previous stage went on to complete 

• Combat Band - Largest volume of 
players completing are in combat band 

000-010 

93.9% / 86529 of completions. 

• Skill Total Band - Largest volume of 
players completing are in skill total band 

0000-0200 

99.88% / 92047 of completions. 

• Starting - average no. players per day 

6947 


• Completing - average no. players per 
day 

4269 

Could have started quest at any time 
previously, not just the date range chosen. 

• Engaging - average no. players per day 

7284 

Could have started quest at any time 
previously, not just the date range chosen. 


Stage completion of participating players 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 

STAGE (LOOKUP TO RIGHT) 

Non-Skip - Players Completing Skip - Players Completing 

Non-Skip - % Complete From Previous Stage 


Completion - Combat vs. Skill Total Mix 


100k 

90k 

80k 

70k 

60k 

50k 

40k 

30k 

20k 

10k 

00 




# # $ 


rS> 

y K y K y 


COMBAT BANDS 


SKILL TOTAL 
BANDS 

■ 2401-2600 

■ 2201-2400 

■ 2001-2200 
■ 1801-2000 
■ 1601-1800 

■ 1401-1600 

■ 1201-1400 

■ 1001-1200 
■ 0801-1000 
■ 0601-0800 
■ 0401-0600 





100% 

- 95% 

- 90% 

- 85% 
80% 
75% 

- 70% 

- 65% 

- 60% 
55% 
50% 

L 22 23 24 



% COMPLETE FROM PREVIOUS STAGE 




% COMPLETE FROM PREVIOUS STAGE 


Tutorial Stage Completion % 
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Launch Week 1 Year Later 



Overall completion rate 

• Launch: 58% 

• 1 year on: 69% 


90% 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 

TUTORIAL STAGE 
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Tools - Pre-production Analytics 

• Churn - Leaver profiling - social connection analysis 


GAME DEVELOPERS CONFERENCE™ EUROPE 2015 AUGUST 3-4, 2015 
Chat connections from engaged players 
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Chat connections from pre-churn players 
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Tools - Pre-production Analytics 

• Churn - Leaver gameplay & chat analysis 
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Tools - Pre-production Analytics 

“What game feature should we work on next?” 

• What problems need to be fixed? 

* Funnel analysis - Locate where player progress is stalling 

* Churn analysis - Identify what leavers have in common 

• What opportunities are open? 

* Content analysis - Identify popular game activities (for a revamp) 

* Player profiling - Find what the most engaged players have in common 


GDCEUROPE.COM 


GAME DEVELOPERS CONFERENCE™ EUROPE 2015 AUGUST 3-4, 2015 

Tools - Pre-production Analytics 


• Content engagement analysis 



Rush 01 Blood 


Shattered Heart 


Skeletal horror 


Wilderness Warbands 


RuneScape Analytics - Content Engagement 

Data taken from 18/08/2014 to 28/05/2015 


Content Category (Select 1 Only) 


Unique Players: 7 Day Average 


Distractions and Diversi 
Product 


Unique Players: 7 Day Average - 142.328 

•it ; of active Daily Growth j Weekly Growth 


Daily Variable Moving Average (Select 1 


01/05/2015 


01/04/2015 01/03/2015 


01/02/2015 01/01/2015 


01/12/2014 


Content Product 


Date 


25/05/2015 24/05/2015 23/05/2015 


22/05/2015 21/05/2015 20/05/2015 


19/05/2015 
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Player Membership w Dodgy? 


Rune Goldberg Machine 


Ironman? 


True - Hardc 


True -Normal 
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000-010 0T1-020 021-030 031-040 


041-050 051-060 061-070 071-080 


081-030 091-100 101- HO 


121-130 131-140 
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Member Content? % 


Grouping System? \ 
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Bork 

Champions’ Challenge 
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Demon Flash Mobs 

Evd Tree 

Familiarisation 

Fish F lingers 

Goblin Flash Mobs 


Statue of Rhiannon 

Tears of Guthw (DnD) 

The Pit 

Troll Invasion 
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Content Category (Select 1 Only) 


Daily Variable Moving Average (Select 1 ... T 

FTEBfffil 7 Day BKW 

Month 

01/05/2015 01/04/2015 01/03/2015 

01/02/2015 01/01/2015 01/12/2014 


Date 


28/05/2015 

27/05/2015 

26/05/2015 

25/05/2015 

24/05/2015 

23/05/2015 

22/05/2015 

21/05/2015 

20/05/2015 

19/05/2015 

18/05/2015 

17/05/2015 

16/05/2015 

15/05/2015 

14/05/2015 


I 


Player Membership w j Dodgy? 

P2P False 

Ironman? 
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T rue - Hardc . . T rue - Normal 


Combat Band 


000-010 

041-050 


on-020 

051-060 


081-030 091-100 


021-030 

061-070 

lovno 


031-040 

071-080 

111-120 


121-130 


131-140 
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Minigame Combat 

Minigame Non-Combat 

Minigame PvP 

Minigame PvP (Safe 1 

Monsters 

Quests 


Training Areas 


Content Product 
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Evil Tree 

Familiarisation 
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God Statues 

Guthixian Caches 

Penguin Hide and Seek 

Phoenix Lair 

Rune Goldberg Machine 

Rush Of Blood 

[ Shattered Heart 

Shooting Star 

Sinkholes 

Skeletal horror 

Statue of Rhiannon 

Tears of Guthix (DnD) 

The Pit 

Troll Invasion 

Wilderness Warbands 



RuneScape Analytics - Content Engagement 

Data taken from 18/08/2014 to 28/05/2015 

Unique Players: 7 Day Average 
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Tools - Pre-production Analytics 

“What game feature should we work on next?” 

• What problems need to be fixed? 

* Funnel analysis - Locate where player progress is stalling 

* Churn analysis - Identify what leavers have in common 

• What opportunities are open? 

* Content analysis - Identify popular game activities (for a revamp) 

* Player profiling - Find what the most engaged players have in common 

• Is this idea viable? 

* Player profiling - Estimate the addressable audience size for a specific feature 

* Content analysis - Examine how similar ideas have performed in the past 
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Tools - Production Analytics 

“How can my designs be as successful as possible?” 

• How should I shape the content? 

• Player profiles - What are the characteristics of the target player? 

• Content analysis - How should I balance this feature? 

• Content analysis - Are the rewards going to be appealing? 



Wlllilill 
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Tools - Production Analytics 

• Player profiling example 
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Tools - Production Analytics 

“How can my designs be as successful as possible?” 

• How should I shape the content? 

• Player profiles - how many active players would qualify? 

• Content analysis - How should I balance this feature? 

• Content analysis - Are the rewards going to be appealing? 

• When should we release it? 

• Seasonality analysis - when are the target audience most engaged? 

• Activity analysis - how long should this event run for? 
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Tools - Production Analytics 

• Seasonality analysis - returner rate 

■ 


Return Rate - Annual Average 
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Tools - Production Analytics 

• Seasonality analysis - daily activity 

Runescape Returners by Day of Week 
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Tools - Post-launch Analytics 

“Did it work as I wanted and how can I make it better?” 

• How is it performing? 

• Content analysis - what is the engagement trend? 

• Does the feature have any defects? 

• Economy analysis - Are the changes in currency/wealth as expected? 

• Quest analysis - Are there places where progression is stalling? 

• How is this feature helping overall? 

• Engagement analysis - Is feature impacting wider player behaviour? 
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Tools - Post-launch Analytics 

• Economy analysis - Loot value 




GAME DEVELOPERS CONFERENCE™ EUROPE 2015 AUGUST 3-4, 2015 

GDCEUROPE.COM 

Tools - Post-launch Analytics 


• iconomy analyses - item trade price and volume 

RurseScapp Analytic* - Grand Exchange »tam History 



Offhand demon stayer mnsbow Oa#y Grand t* chain* trade « 
t price spreads 
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Off-hand demon slayer crossbow: Daily Grand Exchange trade volumes & price spreads 
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Example - Rune Goldberg Machine 

• - design feature to boost rune values 

• Analytics provide balancing data on rune supply and consumption 
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Example - Rune Goldberg Machine 


- assess impact to target metrics 

• Rune prices rise & runecrafting (slightly) increases popularity 


Grand Exchange Market Watch 

Rune Index 


RuneScape XP Gain By Skill 

Sun 01 Mar 2015 to Thu 28 May 2015 
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Game health 

• Metrics for underlying game health 

• Play intensity, play density 
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Game health 

• Play Intensi - average daily play time 
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Game health 

• -days active within a 14 day period 
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Game health 

• Metrics for underlying game health 

• Play intensity, play density 

• Historical benchmarks 

• Divergence from similar period in the past 
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Game health 

• Benchmarking - compare against equivalent historical period 





Average 
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Game health 

• Benchmarking - compare against equivalent historical period 



u 


£ 








& 


v 


& 


JF A# 


* if 


V 


* 




dP ^ 

f X J J J 




GAME DEVELOPERS CONFERENCE™ EUROPE 2015 AUGUST 3-4, 2015 

Game health 

• Metrics for underlying game health 

• Play intensity, play density 


GDCEUROPE.COM 


Historical benchmarks 

• Divergence from similar period in the past 

Customer Satisfaction Surveys 

• Net Promoter Score or Customer Satisfaction Index 
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Game health 

• Customer Satisfaction - Trends over time 


OVERALL SATISFACTION 


Price Increase, 
200 th Quest 

Player Polls, RuneLabs 



Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Feb-15 Apr-15 
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Game health 


• Customer Satisfaction - Split by player cohort 

SATISFACTION COMPARISON FEB VS APR 
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Game health 


• Customer Satisfaction - Feature score & importance 


Feature Survey Q2 2014 Importance Satisfaction 


Look and Feel 
New Content Appeal 
Combat 

Member Attitudes 
Interaction 
Game Information 
Challenges 
PVP 

Treasure Hunter - Value 
Member Sign up Process 
Skills 

Treasure Hunter - Access 
Quests 

Mnmher Sinn l In Product Ranno 
Player Polls 
Com mu n icat ion s 



Customer Satisfaction Index 


72 
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Other Recommendations 

• Have a goal for each project - success criteria 

• Be delicate - don’t make data the enemy! 

• Share data insights across studio 

• Make it data-curiosity part of development culture 

• Balance re-active and pro-active analysis 

• Quick turnaround can be critical 

• Embed analysts and get them involved in creative process 

• Need to fit smoothly into development pipeline 
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